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Executive Summary 


In 2011 and 2012, the National Defense Industrial Association (NDIA) Systems Engineering Di¬ 
vision collaborated with the Institute of Electrical and Electronic Engineers (IEEE) Aerospace and 
Electronic Systems Society and the Carnegie Mellon® Software Engineering Institute (SEI) to 
obtain quantitative evidence of the benefits of systems engineering (SE) best practices on project 
performance. The team developed and executed a survey of system developers to identify SE best 
practices used on projects, collect performance data on these projects, and identify relationships 
between the application of these SE best practices and project performance. Results of this study 
were published in 2012 and 2013 [Elm 2012b, 2013]. The study found clear and significant rela¬ 
tionships between the application of SE best practices to projects and the performance of those 
projects. 

The data collected for the survey was provided by projects operating in the defense domain as 
well as projects from non-defense domains. This distribution of data enabled us to compare and 
contrast both the deployment of SE activities in the two domains and the impact of those SE activ¬ 
ities. Our intent is to identify similarities and differences between the domains that can be instruc¬ 
tive in the effective application of SE. 

We found that the performance {PerJ) of the projects—as measured on a scale of 1 (poor perfor¬ 
mance) to 5 (good performance) based on satisfaction of budget {PerfC), schedule (PerfS), and 
technical requirements {PerfP )—was better for non-defense projects than for defense projects. 
Figure 1 shows the median values for these performance assessments. 



We also examined the degree of project challenge {PC) posed by the projects, measured on a scale 
of 1 (low challenge) to 4 (high challenge). To temper the impact of PC on project performance, 

® Carnegie Mellon is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. 


CMU/SEI-2014-SR-013 | ix 









we adjusted our sample to include only a subsample of defense projects that had the same median 
PC (median = 2.61) as the non-defense projects. 

Finally, we examined the deployment of SE practices in a number of SE process areas. The de¬ 
ployment was measured on a scale of 1 (little SE deployment) to 4 (much SE deployment). Flere, 
we found some differences between the defense and the non-defense domains, as shown in Figure 
2 . 



Figure 2 shows that in nearly all SE process areas except (1) Early SE and (2) Requirements De¬ 
velopment and Management, the SE capabilities {SEC) deployed by the defense projects were 
greater than or equal to those deployed by the non-defense projects. While most of the differences 
appear to be modest, the probability that 10 of the 13 would be better for the non-defense projects 
by chance alone is unlikely (/? < 0.02 using a simple sign test). 

We then examined the relationships between the deployed SE activities and project performance, 
using Goodman and Kruskal’s gamma to quantify the strength of the relationships. A gamma val¬ 
ue near -1 indicates a very strong opposing relationship; a value near 0 indicates a weak or non¬ 
existent relationship; and a value near +1 indicates a very strong supporting relationship. These 
relationships are summarized in Figure 3. 
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Figure 3: Perf vs. SEC Relationship Comparisons 

In Figure 3 and Table 1, we see that 10 of the 13 relationships are stronger for non-defense pro¬ 
jects than they are for defense projects. The probability that 10 of the 13 would be stronger for the 
defense projects by chance alone is, again, unlikely (p < 0.02 using a simple sign test). 


Table 1: Stronger SE Process Areas 


Stronger Perf vs. SEC Relationships 
for Defense Projects 

Stronger Perf vs. SEC Relationships 
for Non-Defense Projects 

• Project Monitoring and Control 

• Trade Studies 

• Integrated Product Teams 

• Total SE 

• Early SE 

• Verification 

• Project Planning 

• Requirements Development and Management 

• Product Integration 

• Validation 

• Configuration Management 

• Product Architecture 

• Risk Management 


We also found a significant difference in the impact of prior experience on project performance, 
with a weak positive relationship (gamma = 0.09) between prior experience {EXP) and Perf for 
the defense-domain projects but a very strong positive relationship (gamma = 0.54) for the non¬ 
defense projects. The difference in the magnitude of the relationship between defense and non¬ 
defense projects indicates that experience has a far greater impact on non-defense projects than it 
has on defense projects. 
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In summary, our data shows that non-defense projects deploy less SE than defense projects, yet 
achieve better performance. We find that the relationships between SE deployment and project 
performance for most SE process areas are stronger for non-defense projects than they are for de¬ 
fense projects. Thus, even though the non-defense projects deploy less SE, they gain more benefit 
from it, contributing to better project performance. 

Of particular consequence are the SE process areas of Product Architecture, Requirements Devel¬ 
opment and Management, and Risk Management. These process areas show notably stronger rela¬ 
tionships for non-defense projects than for defense projects. Understanding the implementation of 
these SE activities in non-defense projects and adopting these practices for defense projects could 
produce significant improvements in the performance of defense-domain projects. 

However, we also note several areas where activities in defense-domain projects show stronger 
relationships to project performance. Defense projects show a considerably stronger relationship 
between Trade Studies and project performance. Additionally, the performance of defense pro¬ 
jects appears to be notably less influenced by project challenge than non-defense projects. Under¬ 
standing the reasons behind these relationships in the defense domain and adopting them for non¬ 
defense projects could produce significant improvements in the performance of non-defense pro¬ 
jects. 

Our data clearly shows the differences in SE deployment and the impact of that SE deployment in 
the defense and non-defense domains. Unfortunately, since our data collection was focused only 
on quantifying deployment and relationships, it cannot explain why these differences exist. 

We plan to perform additional research to further investigate the effectiveness of Product Archi¬ 
tecture, Requirements Development and Management, and Risk Management activities in non¬ 
defense projects and Trade Study activities in defense-domain projects. We will also investigate 
factors that make defense-domain projects less sensitive than non-defense projects to project chal¬ 
lenge. 

The results of these investigations will benefit both domains by enabling them to share best prac¬ 
tices. 
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Abstract 


This report summarizes anaiysis of data coiiected from the 2011 Systems Engineering (SE) Effec¬ 
tiveness Survey performed by the National Defense Industrial Association Systems Engineering 
Division, the Institute of Electrical and Electronics Engineers Aerospace and Electronic Systems 
Society, and the Software Engineering Institute. This analysis examined the differences in the 
deployment and impact of SE activities between defense-domain projects and non-defense pro¬ 
jects. The analysis found significant differences in both the deployment of SE in the two domains 
and the effectiveness of the SE. The report identifies specific process areas where effectiveness in 
one domain is noticeably higher than in the other. Further research to understand these differences 
will benefit both domains by enabling them to share best practices. 
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1 Introduction 


1.1 Purpose 

The purpose of this report is to compare and contrast the impact of systems engineering (SE) ac¬ 
tivities on projects in the defense domain with projects in other commercial domains. 

1.2 Background 

1.2.1 Developing the SE Effectiveness Survey 

In 2011 and 2012, the National Defense Industrial Association (NDIA) Systems Engineering Di¬ 
vision, the Institute of Electrical and Electronic Engineering (IEEE) Aerospace and Electronic 
Systems Society, and the Software Engineering Institute (SEI) collaborated to perform the Sys¬ 
tems Engineering Effectiveness Study. The purpose of this study was to identify the relationship 
between the application of SE activities to projects and the performance of those projects. Such 
data would promote an understanding of the value of SE and build a business case for SE that 
would justify a project’s investment in SE resources and activities. 

The study was performed by surveying 148 individual projects to determine 

• the SE activities applied to individual projects 

• the performance of those projects as measured by satisfaction of budget, schedule, and tech¬ 
nical requirements 

• factors other than SE that could impact performance, such as 

the degree of challenge posed by the project 

the environment in which the project was executed 

prior experience of the organization and the team executing the project 

To assess the SE activities applied to the project, we questioned the respondents about work prod¬ 
ucts resulting from specific SE activities. The survey questionnaire assessed SE deployment via 
82 questions addressing the presence and quality of 51 work products (see Appendix A). 

We assessed project performance in terms of meeting schedule, meeting budget, and satisfying 
technical requirements. We collected multiple project performance measures and looked for the 
degree of agreement between these measures. An in-depth discussion of the collection and analy¬ 
sis of project performance information is found in the report The Business Case for Systems Engi¬ 
neering Study: Assessing Project Performance from Sparse Data [Elm 2012a]. 

We also collected information on several other factors that can impact project performance, in¬ 
cluding the degree of challenge posed by the project, the environment in which the project was 
executed, and prior experience of the organization and the team executing the project. 
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1.2.2 Executing the SE Effectiveness Survey 


To reach project members to participate in the survey, we relied on the resources of the NDIA, 
IEEE, and the International Council on Systems Engineering (INCOSE). The sample is not based 
on known probability of selection criteria, so the results related to the distributions of any single 
variable cannot be generalized to the entire population of system developers. However, there is 
sufficient variation in the sample to analyze relationships among those variables. It is those rela¬ 
tionships that allow us to address the validity of assertions about the effects that SE activities can 
have on program performance under varying circumstances. 

We made the process of responding to the survey simple and convenient by providing the survey 
online. To participate, a respondent merely had to obtain an online account from the survey serv¬ 
er, log in, and complete the survey. 

Some organizations were reluctant to respond due to the survey’s request for competition- 
sensitive information identifying project performance. To mitigate these concerns, we collected 
all responses anonymously and did not solicit information to identify people, projects, or organi¬ 
zations. Additionally, we promised that data would be collected and handled by a trasted organi¬ 
zation—the SEI, which is a federally funded research and development center (FFRDC) that does 
not compete with any of the responding organizations and is known for handling sensitive data. 

Survey data collection occurred between October 2011 and March 2012. Valid survey responses 
were received from 148 projects. 

1.2.3 Survey Data Analysis 

The primary survey hypotheses were that the deployment of SE best practices, the degree of chal¬ 
lenge imposed by the project, and the environment in which the project was executed all have a 
quantifiable impact on project performance. This can be stated mathematically as 

Perf^ f{PQ EXP, SEC), 

where project challenge prior experience 

SEC = systems engineering capability Perf= project performance 

Our goal was to identify the impact of PC, EXP, and SEC on Perf. We did this by first scoring 
each parameter based on the survey responses and then by identifying the relationships among 
these scores. 

1.2.3.1 Scoring 

Project performance {Perf) was measured based on three subfactors: cost performance (PerfC), 
schedule performance (PerfS), and technical performance {PerfP). These subfactors were com¬ 
bined to create a measure of overall performance {Perf that ranged in value from 1 (very poor 
performance) to 5 (very good performance). 

Systems Engineering Capability {SEC) is a measure of the SE activities applied to each project 
and was assessed based on the presence and quality of the 51 SE work products listed in Appen¬ 
dix A. Because these work products could be categorized within specific areas of SE, we could 
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assess not only the total SE activities (SEC-Total) applied to the project but also the SE capabili¬ 
ties in the following SE process groups: 


Project Planning 
Verification 

Project Monitoring and Control 

Requirements Development and Management 

Trade Studies 

Validation 

Product Integration 

Configuration Management 

Product Architecture 

Integrated Product Teams (IPT)-based capability 
Risk Management 


(SEC-PP) 

(SEC-VER) 

{SEC-PMQ 

(SEC-REQ) 

(SEC-TRD) 

(SEC-VAL) 

(SEC-PI) 

(SEC-CM) 

(SEC-ARCH) 

(SEC-IPT) 

(SEC-RSKM) 


Each of these factors, as well as SEC-Total, was assessed over the range of 1 (low capability) to 4 
(high capability). 

Project challenge (PC) was assessed via questions probing a number of diverse issues such as pro¬ 
ject size, project complexity, and technology precedents. All of these factors are combined into a 
single PC measure scaled from 1 (not very challenging) to 4 (very challenging). 

Other factors with the potential to impact project performance were also assessed, including prior 
experience, contract type, SE organization, percentage of project completion, and project SE con¬ 
tent. 

1.2.3.2 Identifying Relationships 

Once the scores of all parameters were calculated, we identified relationships between the scores. 
Relationships were evaluated and presented in two ways: 

1. via mosaic charts 

2. via nonparametric statistical analysis 

Mosaic charts provide an intuitive and visual means of examining the relationship between a de¬ 
pendent variable (e.g., Perf depicted on the vertical axis) and an independent variable (e.g., SEC- 
Total or SEC-PP depicted on the horizontal axis). Development of a mosaic chart starts with the 
scores of the independent and dependent variable (i.e., SEC-Total and Perf). For each project, the 
scores are categorized as lower, middle, or higher, based on the establishment of breakpoints that 
distribute the projects evenly between these categories. The categorized scores are then tallied and 
displayed on the mosaic chart to show the relationship between the variables, as illustrated in Fig¬ 
ure 4. In this case, it is visually evident that those programs with higher SE deployment tend to 
deliver higher project performance. 


CMU/SEI-2014-SR-013 | 3 



Percentage of projects 
delivering HIGHER 
levels of performance 


Percentage of projects 
delivering MIDDLE 
levels of performance 


Percentage of projects 
delivering LOWER 
levels of performance 


Project Performance vs. Total SE 



Projects 

deploying 

LOWER 

levels of 

systems 

engineering 


Projects 

deploying 

MIDDLE 

levels of 

systems 

engineering 


□ Projects delivering 
HIGHER 
performance 

□ Projects delivering 
MIDDLE 
performance 

□ Projects delivering 
LOWER 
performance 


Projects 

deploying 

HIGHER 

levels of 

systems 

engineering 


Figure 4: Mosaic Chart Key 


Figure 4 also contains a nonparametric statistical measure of the relationship between SEC-Total 
and Perf. Goodman and Kmskal’s gamma is a measure of association that expresses the strength 
of relationship between two ordinal variables. Gamma varies from +1 to -1, with values near -1 
indicating a strong opposing relationship, values near 0 indicating a weak or no relationship (sta¬ 
tistical independence), and values near +1 indicating a strong supporting relationship. 

1.2.4 Initial Results of the SE Effectiveness Survey Results 

We received 148 valid survey responses, with response demographics as shown in Figure 5. Both 
the distributions for SE deployment {SEC-Total) and project performance {Perf) are skewed to the 
right, indicating that the surveyed sample consisted of projects deploying larger amounts of SE 
and producing higher levels of performance. Some of this bias may be due to the inherent opti¬ 
mism of self-reporting. Fortunately, the analysis techniques utilized to identify the relationships 
between the independent and dependent variables are not affected by this bias. 
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Istquartile 3.03 
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PC 


1st quartile 2.22 
Median (2nd quartile) 2.50 
3rd quartile 2.68 


Contract Value 


Mean 488 M$ 
Std. Dev. 2,220 MS 
Median 50.5 M$ 




Figure 5: Response Demographics 

The study found strong relationships between the deployment of SE and project performance, as 
shown in Figure 6. 


Project Performance vs. Total-SEC 



□ Projects delivering 
HIGHER 
performance 

□ Projects delivering 
MIDDLE 
performance 

ED Projects delivering 
LOWER 
performance 


Total Systems Engineering Capability {SEC) 

gamma = 0.49 


Figure 6: Project Performance vs. Totai SE Capabiiity 


This figure clearly shows that projects that deploy more SE deliver better performance. This rela¬ 
tionship is amplified for projects posing more challenge, as shown in Figure 7. 
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Figure 7: Project Performance vs. Total SE Capability Controlled by Project Challenge 

The individual process areas of SE and project performance display similarly strong relationships. 
Figure 8 shows the strength of the relationships between each of the factors and project perfor¬ 
mance, as measured by gamma. 

Performance vs. SE Capability - All Projects 

- 0.3 - 0.2 - 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 

Total SE 
Early SE 
Project Planning 
Req'ts Dev't & Mg't 
Verification 
Product Architecture 
Configuration Mg't 
Trade Studies 
Monitor & Control 
Validation 
Product Integration 
Risk Management 
Integ. Product Teams 
Project Challenge 
Prior Experience 

Moderate Weak Moderate Strong Very Strong 



Figure 8: Project Performance vs. SE Deployment 

Significantly, all of the SE factors have a positive relationship with project performance, and for 
most of them that relationship is either strong or very strong. 

The results of this study are presented in considerable detail in 

• The Business Case for Systems Engineering Study: Results of the Systems Engineering Effec¬ 
tiveness Survey [Elm 2012b] 

• The Business Case for Systems Engineering Study: Detailed Response Data [Elm 2013] 
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1.3 Defense vs. Non-Defense Projects 


In conducting the SE Effectiveness Survey in 2011 and 2012, we solicited project members 
through the resources of NDIA, IEEE, and INCOSE. While many of those who participated in the 
survey were associated with the U.S. defense domain, a nontrivial number were not associated 
with defense-related activities. This division is shown in Figure 9. 
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Figure 9: Industry and End User 

Figure 9 shows that 116 respondents declared their primary industry as defense related, while 31 
respondents declared their primary industry as non-defense related. These non-defense projects 
supported industrial domains including energy, communications, and health care. The figure also 
shows that 115 respondents declared that the end user for their products was a defense organiza¬ 
tion, while 32 declared that the end user was not a defense organization.* 

Since we could categorize the responding projects as either defense related or non-defense related, 
we could analyze each group independently to identify relationships between SE deployment, 
project challenge, experience, and project performance. Independent analysis enabled us to com¬ 
pare and contrast these groups to identify similarities and differences that can be instructive in the 
effective application of SE. 


In total, 148 responses were received. The numbers cited here may not add to 148 because some respondents 
chose not to answer the questions identifying their primary industry and end users. 
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2 Comparison of Projects in the Defense and Non-Defense 
Domains 

2.1 Project Characteristics 

Before comparing and contrasting the SE and performance characteristics of the two project 
groups, we must first understand the demographics of the groups. 

Figure 10 shows the distribution of project contract values for the projects in the defense domain 
and those not in the defense domain. The median contract value for the group of defense projects 
was $60 million, while the median contract value for the group of non-defense projects was $40 
million. While this represents a 33% difference, the two sets of projects are within the same order 
of magnitude. 



Figure 10: Contract Value 

Figure 11 shows the distribution of project duration for the groups of projects in the defense and 
non-defense domains. The median project duration for the group of defense projects was 48 
months, while the median project duration of the non-defense projects was only 10% less at 44 
months. 



Figure 11: Contract Duration 
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We did not restrict our surveys to completed projects; thus our survey responses included projects 
at various stages of completion. Figure 12 shows the distribution of the completion status of the 
groups of projects in the defense and non-defense domains. Within the defense domains, the mean 
completion status of the projects is 71%, and in the non-defense domain, the mean is nearly the 
same at 72%. 
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Figure 12: Current Completion Status 


Starting projects before they are sufficiently defined can impact project performance. One meas¬ 
ure of the quality of the project definition is the completeness of the customer requirements. Fig¬ 
ure 13 shows the percentage of undefined customer requirements at the time of contract award. 



Figure 13: Percentage of Customer Technical Requirements Marked “TBD” at Contract Award 

For the group of defense-domain projects, half of the projects have <10% undefined requirements 
at the time of award. The group of non-defense projects has the same median. The mean value for 
the percentage of undefined requirements in the defense domain is 16%, compared to a mean val¬ 
ue for the non-defense domain of 22%. This indicates that initial project definition in the non¬ 
defense domain may be less complete than that within the defense domain. 

As the project progresses, many of the customer requirements that were initially undefined be¬ 
come known. Figure 14 shows the percentage of undefined technical requirements within the pro¬ 
jects at the time of the survey. 
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Figure 14: Percentage of Customer Technical Requirements Currently Marked “TBD” 

Within the groups of defense and non-defense projects, over half of the projects in each group had 
no undefined customer requirements at the time of the survey, as evidenced by the median values 
of 0%. The 10% mean percentage of undefined requirements for the non-defense projects is only 
slightly higher than the mean of 7% for the defense projects, indicating no significant difference 
in project definition between the two groups. 

In most cases, SE is only a part of the total non-recurring engineering (NRE) applied to a project. 
Figure 15 shows the amount of SE applied to a project as a percentage of the total NRE of the 
project. The defense-domain projects and the non-defense projects show little difference between 
the median (15% vs. 13%) and mean (21% vs. 26%) values for the SE percentage. 



Figure 15: SE Percentage of Non-Recurring Engineering 

2.2 Project Challenge 

As discussed previously, we assessed the degree of challenge posed by each project by evaluating 
factors such as project size, project complexity, and technology precedents. The distribution of 
project challenge (PC) is shown in Figure 16, with the value of 1 representing very low challenge 
and 4 representing very high challenge. 
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PC (Defense) Istquartile 2.17 

Median (2nd quartile) 2.44 
3rd quartile 2.67 
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Figure 16: PC Response Distribution 

For the defense-domain projects, the median of 2.44 indicates that the sampled projects were 
nearly in the middle of the continuum of project challenge. Similar conclusions may be drawn for 
the non-defense projects, with a median of 2.61, which appeared to be a bit more challenging than 
the defense projects. 

As part of preparing the mosaic chart showing the relationship between PC and Perf, three groups 
for PC were established with breakpoints at 2.33 and 2.65. These breakpoints resulted in a nearly 
even distribution of projects across the three categories of lower challenge, middle challenge, and 
higher challenge. The resulting mosaic charts are shown in Figure 17. 
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Figure 17: PC vs. Perf 

These charts reveal a moderately negative relationship (gamma = -0.21) between PC and Perf for 
the defense-domain projects, but a very strong negative relationship (gamma = -0.44) for the non- 
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defense projects. The negative polarity of these relationships is consistent with intuition—one 
would expect more challenging projects to have greater difficulty achieving performance expecta¬ 
tions. The difference in the magnitude of the relationship between defense and non-defense pro¬ 
jects is interesting and will be discussed later in this report. 

2.2.1 Compensating for Differences in Project Chaiienge 

To achieve the best comparison between defense and non-defense projects, to the greatest extent 
possible, we endeavored to eliminate variations in other factors affecting project performance. PC 
is one of those factors. The 116 projects in the defense domain sample had a median PC value of 
2.44, while the 32 non-defense projects had a median PC value of 2.66, representing a higher de¬ 
gree of challenge. 

To temper the impact of this variation in PC, we identified a subset of the defense projects that 
also had a median PC value of 2.66. This subset consisted of the 75 most challenging defense pro¬ 
jects. As shown in Figure 18, the PC distributions for the defense and non-defense projects are 
remarkably similar. 



For the remainder of this report, we will perform all analyses using this 75-project subset of the 
defense-domain projects. 

The resulting mosaic charts of Figure 19 show the relationship between the PC subset and Perf. 
We established three groups for PC with breakpoints at 2.45 and 2.70 to obtain a nearly even dis¬ 
tribution of projects across the three categories of lower challenge, middle challenge, and higher 
challenge. These charts reveal a weak negative relationship (gamma = -0.08) between PC and 
Perf for the defense-domain projects, but a moderate negative relationship (gamma = -0.24) for 
the non-defense projects. 
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Figure 19: Adjusted PC i/s. Perf 

2.3 Project Performance 

As discussed previously, the performance (Perf) of each project was assessed by evaluating its 
conformance to budget (PerfC), schedule (PerfS), and technical (PerfP) requirements. These three 
factors were combined additively to create a measure of overall project performance. Distribution 
of Perf for defense and non-defense projects is shown in Figure 20. 


Per/(Defense) Istquartile 2.97 

Median {2nd quartile) 3.51 



Per/(Non-Defense) istquartile 2.92 

Median (2nd quartile) 3.59 
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Figure 20: Project Performance Response Distribution 

A Perf value of 3.00 may be interpreted as a project that is on schedule, on budget, and satisfying 
technical requirements. Values greater than 3 indicate superior performance (e.g., under budget, 
ahead of schedule, and/or exceeding technical requirements), while values less than 3 represent 
inferior performance. The median values of 3.51 for the defense projects and 3.59 for the non- 
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defense projects indicate that the projects surveyed, in the aggregate, exceeded performance ex¬ 
pectations in terms of cost, schedule, and/or technical performance. 


Figure 21 shows the distributions of the performance subfactors of cost performance, schedule 
performance, and technical performance. 
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Figure 21: Performance Sub factor Distributions 


Only the distribution of the schedule subfactor (PerfS) shows a noticeable difference between the 
defense projects (median = 3.50) and the non-defense projects (median =3.58). All of these per¬ 
formance measures are shown graphically in Figure 22. 
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P erf is used as the primary variable for comparison with SEC variables. As discussed previously, 
the first step in preparing the mosaic charts showing these relationships is to divide the projects 
into three groups based on Perf -—^those delivering lower, middle, and higher performance. We do 
this by examining the cumulative distribution function of Perf (Figure 23) to identify two break¬ 
points that distribute the projects evenly into three bins.^ 

Perf 

120 % 

100 % 

80% 

60% 

40% 

20 % 

0 % 

1 1.5 2 2.5 3 3.5 4 4.5 5 

Figure 23: Perf Cumulative Response Distribution 



Examining the data, we can identify these breakpoints at 3.18 and 3.83. Thus, for all future analy¬ 
sis, we categorize projects as follows: 


1 < Perf < 3.18 

3.18 < Perf < 3.83 

3.83 < Perf 


Lower project performance 
Middle project performance 
Fligher project performance 


More detail about establishing the cutting points can be found in The Business Case for Systems Engineering 
Study: Assessing Project Performance from Sparse Data [Elm 2012a]. 
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2.4 Systems Engineering Capabilities 

The following sections discuss the results of the survey for SE capabilities. 

2.4.1 Systems Engineering Process Areas 

2.4.1.1 Project Planning 

The distributions of the responses assessing Project Planning activities (SEC-PP) are shown in 
Figure 24, with a value of 1 representing very poor Project Planning and 4 representing very good 
Project Planning. The respective median values of 2.96 and 2.87 for the defense and non-defense 
projects indicate that defense projects deploy somewhat more Project Planning activities than 
non-defense projects. 



The mosaic charts of Figure 25 show the relationship between SEC-PP and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-PP with breakpoints at 2.82 and 3.25 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.57) between SEC-PP and Perf for the defense projects and a very strong 
supporting relationship (gamma = 0.59) for the non-defense projects. 
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Figure 25: SEC-PP vs. Perf 

2.4.1.2 Verification 

The distributions of the responses assessing Verification activities (SEC-VER) of the projects are 
shown in Figure 26, with a value of 1 representing very poor Verification work and 4 representing 
very good Verification work. The respective median values of 3.11 and 3.00 for the defense and 
non-defense projects indicate that defense projects deploy somewhat more Verification activities 
than non-defense projects. 
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Figure 26: SEC-VER Response Distribution 

The mosaic charts of Figure 27 show the relationship between SEC- VER and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-VER with breakpoints at 2.88 and 3.40 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela- 
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tionship (gamma = 0.55) between SEC-VER and Perf for the defense projects and a very strong 
supporting relationship (gamma = 0.64) for the non-defense projects. 

Perf vs. SEC-VER (Defense) Perf vs. SEC-VER (Non-Defense) 
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Figure 27: SEC-VER vs. Perf 


2.4.1.3 Project Monitoring and Controi 

The distributions of the responses assessing Project Monitoring and Control activities (SEC-PMC) 
of the projects are shown in Figure 28, with a value of 1 representing very poor Project Monitor¬ 
ing and Control and 4 representing very good Project Monitoring and Control. The respective me¬ 
dian values of 3.09 and 3.00 for the defense and non-defense projects indicate that defense pro¬ 
jects deploy somewhat more Project Monitoring and Control activities than non-defense projects. 
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Figure 28: SEC-PMC Response Distribution 

The mosaic charts of Figure 29 show the relationship between SEC-PMC and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
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SEC—for SEC-PMC with breakpoints at 2.90 and 3.30 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.48) between SEC-PMC and Perfior the defense projects and a strong sup¬ 
porting relationship (gamma = 0.38) for the non-defense projects. 
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Figure 29: SEC-PMC vs. Perf 


2.4.1.4 Requirements Development and Management 

The distributions of the responses assessing the Requirements Development and Management 
activities (SEC-REQ) of the projects are shown in Figure 30, with a value of 1 representing very 
poor Requirements Development and Management and 4 representing very good Requirements 
Development and Management. The respective median values of 3.15 and 3.18 for the defense 
and non-defense projects indicate no significant difference in the deployment of Requirements 
Development and Management activities between the two project groups. 
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Figure 30: SEC-REQ Response Distribution 
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The mosaic charts of Figure 31 show the relationship between SEC-REQ and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-REQ with breakpoints at 2.90 and 3.45 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of this chart reveals a very strong supporting rela¬ 
tionship (gamma = 0.46) between SEC-REQ and Perf for the defense projects, and a very strong 
supporting relationship (gamma = 0.60) for the non-defense projects. 
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Figure 31: SEC-REQ vs. Perf 

2.4.1.5 Trade Studies 

The distributions of the responses assessing Trade Study activities (SEC-TRD) of the projects are 
shown in Figure 32, with a value of 1 representing very poor Trade Study work and 4 representing 
very good Trade Study work. The respective median values of 3.00 and 3.00 for the defense and 
non-defense projects indicate no difference in the deployment of Trade Study activities between 
the two project groups. 
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Figure 32: SEC-TRD Response Distribution 
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The mosaic charts of Figure 33 show the relationship between SEC-TRD and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-TRD with breakpoints at 2.70 and 3.00 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.45) between SEC-TRD and Perf for the defense projects but only a strong 
supporting relationship (gamma = 0.30) for the non-defense projects. 
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Figure 33: SEC-TRD vs. Perf 

2.4.1.6 Validation 

The distributions of the responses assessing Validation activities {SEC-VAL) of the projects are 
shown in Figure 34, with a value of 1 representing very poor Validation work and 4 representing 
very good Validation work. The respective median values of 3.00 and 3.00 for the defense and 
non-defense projects indicate no apparent difference in the deployment of Validation activities 
between defense and non-defense projects. 
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The mosaic charts of Figure 35 show the relationship between SEC- VAL and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-VAL with breakpoints at 2.50 and 3.00 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.45) between SEC- VAL and Perf for the defense projects and a very strong 
supporting relationship (gamma = 0.47) for the non-defense projects. 
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Figure 35: SEC-VAL vs. Perf 

2.4.1.7 Product Integration 

The distributions of the responses assessing Product Integration activities (SEC-PI) of the projects 
are shown in Figure 36, with a value of 1 representing very poor Product Integration and 4 repre¬ 
senting very good Product Integration. The respective median values of 3.00 and 3.00 for the de- 
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fense and non-defense projects indicate no difference in the deployment of Product Integration 
activities between the two project groups. 
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Figure 36: SEC-PI Response Distribution 

The mosaic charts of Figure 37 show the relationship between 5'£’C-/’/and Pet/for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-PI with breakpoints at 2.50 and 3.50 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.45) between SEC-PI and Perf for the defense projects and a very strong sup¬ 
porting relationship (gamma = 0.52) for the non-defense projects. 


Perf vs. SEC-PI (Defense) 


Perf vs. SEC-PI (Non-Defense) 


90% - 

17% 

33% 


80% - 

70% - 

25% 

47% 

60% - 







50% - 

40% - 


33% 


30% - 

20% - 

10% - 

0% - 

58% 

I) 33% ;|; 

47% 

b7%d 


□ Projects 
delivering 
HIGHER 
performance 

□ Projects 
delivering 
MIDDLE 
performance 

□ Projects 
delivering 
LOWER 
performance 


Lower SEC Middle SEC Higher SEC 
(n=12) (n=48) (n=15) 



100 % 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20 % 

10 % 

0 % 


Lower SEC Middle SEC Higher SEC 
(n=9) (n=14) (n=8) 


SE Deployment 


gamma = 0.45 


SE Deployment 


gamma = 0.52 


Figure 37.- SEC-PI vs. Perf 
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2.4.1.8 Configuration Management 

The distributions of the responses assessing Configuration Management activities (SEC-CM) of 
the projects are shown in Figure 38, with a value of 1 representing very poor Configuration Man¬ 
agement and 4 representing very good Configuration Management. The respective median values 
of 3.40 and 3.20 for the defense and non-defense projects indicate that defense projects deploy 
more Configuration Management activities than non-defense projects. 
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Figure 38: SEC-CM Response Distribution 

The mosaic charts of Figure 39 show the relationship between 5EC-CM and Pe//for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-CM with breakpoints at 3.10 and 3.80 fo creafe a nearly even disfribufion of pro- 
jecfs across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.42) between SEC-CM and Perf for the defense projects and a very strong 
supporting relationship (gamma = 0.47) for the non-defense projects. 
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Figure 39: SEC-CM i/s. Perf 


2.4.1.9 Product Architecture 

The distributions of the responses assessing Product Architecture activities {SEC-ARCH) of the 
projects are shown in Figure 40, with a value of 1 representing very poor Product Architecture 
work and 4 representing very good Product Architecture work. The respective median values of 
3.20 and 3.00 for the defense and non-defense projects indicate that the defense projects deploy 
more Product Architecture activities than the non-defense projects. 
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Figure 40: SEC-ARCH Response Distribution 

The mosaic charts of Figure 41 show the relationship between SEC-ARCH and Perf for the de¬ 
fense and non-defense projects. We established three categories—lower SEC, middle SEC, and 
higher SEC—for SEC-ARCH with breakpoints at 2.95 and 3.40 to create a nearly even distribu¬ 
tion of projects across the three categories. Examination of these charts reveals a strong support- 
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ing relationship (gamma = 0.38) between SEC-ARCH and Perf for the defense projects and a very 
strong supporting relationship (gamma = 0.54) for the non-defense projects. 
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Figure 41: SEC-ARCH vs. Perf 

2.4.1.10 Integrated Product Teams 

The distributions of the responses assessing the use of Integrated Product Teams (SEC-IPT) are 
shown in Figure 42, with a value of 1 representing very poor use and 4 representing very good 
use. The respective median values of 3.00 and 3.00 for the defense and non-defense projects indi¬ 
cate no apparent differences in the deployment of Integrated Product Teams activities between the 
two project groups. 
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Figure 42: SEC-IPT Response Distribution 
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The mosaic charts of Figure 43 show the relationship between SEC-IPT and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-IPT with breakpoints at 2.6 and 3.00 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a strong supporting relation¬ 
ship (gamma = 0.32) between SEC-IPT and Pe//for the defense projects and a moderate support¬ 
ing relationship (gamma = 0.23) for the non-defense projects. 
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Figure 43: SEC-IPT vs. Perf 

2.4.1.11 Risk Management 

The distributions of the responses assessing Risk Management activities {SEC-RSKM) of the pro¬ 
jects are shown in Figure 44, with a value of 1 representing very poor Risk Management and 4 
representing very good Risk Management. The respective median values of 3.00 and 2.88 for the 
defense and non-defense projects indicate that defense projects deploy somewhat more Risk Man¬ 
agement activities than non-defense projects. 
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Figure 44: SEC-RSKM Response Distribution 
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The mosaic charts of Figure 45 show the relationship between SEC-RSKM and Perf for the de¬ 
fense and non-defense projects. We established three categories—lower SEC, middle SEC, and 
higher SEC—for SEC-RSKM with breakpoints at 2.75 and 3.40 to create a nearly even distribu¬ 
tion of projects across the three categories. Examination of these charts reveals a moderate sup¬ 
porting relationship (gamma = 0.28) between SEC-RSKM and Perf for the defense projects and a 
very strong supporting relationship (gamma = 0.40) for the non-defense projects. 
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Figure 45: SEC-RSKM vs. Perf 

2.4.2 Composite Systems Engineering Capabiiities 

2.4.2.1 Total Systems Engineering Capability 

As discussed previously, the total SE capability (SEC-Total) applied to a project is assessed as the 
amalgam of SE capabilities applied in the following process groups: 

• Project Planning (SEC-PP) 

• Verification (SEC- VER) 

• Project Monitoring and Control (SEC-PMC) 

• Requirements Development and Management (SEC-REQ) 

• Trade Studies (SEC-TRD) 

• Validation (SEC- VAL) 

• Product Integration (SEC-PI) 

• Configuration Management (SEC-CM) 

• Product Architecture (SEC-ARCH) 

• Integrated Product Teams (SEC-IPT) 

• Risk Management (SEC-RSKM) 
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The distributions of SEC-Total are shown in Figure 46, with a value of 1 representing very poor 
SE deployment and 4 representing very good SE deployment. The median values of 3.05 and 3.00 
for the defense and non-defense projects, respectively, show that defense-domain projects deploy 
slightly more SE than non-defense projects. 
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Figure 46: SEC-Total Response Distribution 

The mosaic charts of Figure 47 show the relationship between SEC-Total and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC—for SEC-Total with breakpoints at 2.87 and 3.27 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.57) between SEC-Total and Peif for the defense projects. But the relation¬ 
ship between SEC-Total and Perf for the non-defense projects is even stronger, with a gamma 
value of 0.66. 
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Figure 47: SEC-Total vs. Perf 
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2.4.2.2 Early Systems Engineering 


One of our hypotheses was that SE activities performed early in the program life cycle have the 
greatest impact on project performance. If this hypothesis is hue, then we would expect to find the 
strongest relationships between these early SE activities and project performance. SE activities 
that are predominantly performed early in the system life cycle include 

• Project Planning 

• Requirements Development and Management 

• Trade Study performance 

• Product Architecture development 

We formed an amalgam of these SE process areas, labeled it Early SE, and analyzed its relation¬ 
ship to project performance. 


The distributions of the responses assessing Early SE are shown in Figure 48, with a value of 1 
representing very poor use and 4 representing very good use. The respective median values of 
3.03 and 3.06 for the defense and non-defense projects show that defense-domain projects deploy 
slightly less Early SE than non-defense projects. 
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Figure 48: Early SE Response Distribution 

The mosaic charts of Figure 49 show the relationship between Early SE and Perf for the defense 
and non-defense projects. We established three categories—lower SEC, middle SEC, and higher 
SEC —for Early SE with breakpoints at 2.85 and 3.375 to create a nearly even distribution of pro¬ 
jects across the three categories. Examination of these charts reveals a very strong supporting rela¬ 
tionship (gamma = 0.61) betweenE’ar/y SE andPerfior the defense projects and a very strong 
supporting relationship (gamma = 0.62) for the non-defense projects. 
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Figure 49: Early SE i/s. Perf 

2.5 Experience 

The prior experience of the project team and the organization can have an impact on project per¬ 
formance. Figure 50 shows the distributions of survey responses assessing the prior experience 
metric (EXP) for the defense and the non-defense projects. The two groups of projects show little 
difference in experience. 
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Figure 50: Experience Response Distribution 


The mosaic charts of Figure 51 show the relationship between EXP and Perf for the defense and 
non-defense projects. Three categories—lower experience, middle experience, and higher experi¬ 
ence—were established ior EXP with breakpoints at 2.9 and 3.3 to create a nearly even distribu¬ 
tion of projects across the three categories. These charts reveal a weak positive relationship 
(gamma = 0.09) between EXP and Perf for the defense-domain projects, but a very strong positive 
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relationship (gamma = 0.54) for the non-defense projects. The difference in the magnitude of the 
relationship between defense and non-defense projects indicates that experience has a far greater 
impact on non-defense projects than it has on defense projects. 
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Figure 51: Performance vs. Experience 
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3 Summary and Conclusions 


3.1 Summary 

The impetus for this survey was a desire to compare and contrast the application and impact of SE 
in two different domains—defense and non-defense—with the hope of identifying best practices 
in one domain that could be transferred to the other to further improve project performance. Our 
study found significant differences in both the deployment of these practices and the impact of 
these practices on project performance. 

Our defense domain sample consisted of 116 project participants who self-identified as defense 
related. Of these defense projects, 75 were selected based on their PC score to create a sample 
reflecting degrees of project challenge equivalent to the sample of non-defense projects. Our non¬ 
defense sample consisted of 32 projects from sectors such as electronic equipment, communica¬ 
tions, and health care. 

We examined the performance of the projects, as measured on a scale of 1 (poor performance) to 
5 (good performance) based on satisfaction of budget, schedule, and technical requirements. We 
found that performance against budget and technical requirements was essentially the same in 
both the defense and the non-defense domains. However, non-defense projects performed some¬ 
what better against schedule than did defense projects. Table 2 and Figure 52 show the median 
values of the assessed performance measures. 


Table 2: Project Performance Comparison 


Parameter 

Median Values 

Defense 

Projects 

Non-Defense 

Projects 

Total performance (Perf) 

3.51 

3.59 

Cost performance [PerfC) 

3.50 

3.50 

Schedule performance {PerfS) 

3.50 

3.58 

Technical performance (PerfT) 

3.67 

3.67 
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We examined the degree of challenge posed by the projects, measured on a scale of 1 (low chal¬ 
lenge) to 4 (high challenge). Because we selected a subsample of defense projects to have the 
same median challenge as the non-defense projects, both groups exhibited a median value of 2.61 
for PC. 

Finally, we examined the deployment of SE practices in a number of SE process areas. The de¬ 
ployment was measured on a scale of 1 (little SE deployment) to 4 (much SE deployment). Flere, 
we found some differences between the defense and the non-defense domain, as shown in Table 3 
and Figure 53. 


Table 3: SE Deployment Comparison 


Parameter 

Median Values 

Defense Projects 

Non-Defense Projects 

Total SE 

3.05 

3.00 

Early SE 

3.03 

3.06 

Project Planning 

2.96 

2.87 

Verification 

3.11 

3.00 

Monitoring and Control 

3.09 

3.00 

Requirements Dev’t & Mg’t 

3.15 

3.18 

Trade Studies 

3.00 

3.00 

Validation 

3.00 

3.00 

Product Integration 

3.00 

3.00 

Configuration Management 

3.40 

3.20 

Product Architecture 

3.20 

3.00 

Integrated Product Teams 

3.00 

3.00 

Risk Management 

3.00 

2.88 
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In nearly all SE process areas except (1) Early SE and (2) Requirements Development and Man¬ 
agement, the SE capabilities deployed by the defense projects were greater than or equal to that 
deployed by the non-defense projects. While most of the differences appear to be modest, the 
probability that 10 of the 13 would be greater for the non-defense projects by chance alone is un¬ 
likely {p < 0.02 using a simple sign test) [Kitchens 2002]. 

We then examined the relationships between the deployed SE activities and project performance, 
using Goodman and Kruskal’s gamma to quantify the strength of the relationship. A gamma value 
near -1 indicates a very strong opposing relationship; a value near 0 indicates a weak or nonexist¬ 
ent relationship; and a value near +1 indicates a very strong supporting relationship. These rela¬ 
tionships are summarized in Table 4 and Figure 54. 


CMU/SEI-2014-SR-013 | 35 








Table 4: Perf vs. SEC Relationship Comparisons 


Parameter 

Gamma 

Defense Projects 

Non-Defense Projects 

Perf vs. Total SE 

0.57 

0.66 

Perf vs. Early SE 

0.61 

0.62 

Perf vs. Project Challenge 

-0.08 

-0.24 

Perf vs. Project Planning 

0.57 

0.59 

Perf vs. Verification 

0.55 

0.64 

Perf vs. Monitoring and Control 

0.48 

0.38 

Perf vs. Requirements Dev’t & Mg’t 

0.46 

0.60 

Perf vs. Trade Studies 

0.45 

0.30 

Perf vs. Validation 

0.45 

0.47 

Perf vs. Product Integration 

0.45 

0.52 

Perf vs. Configuration Management 

0.42 

0.47 

Perf vs. Product Architecture 

0.38 

0.54 

Perf vs. Integrated Product Teams 

0.32 

0.23 

Perf vs. Risk Management 

0.28 

0.40 
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Figure 54: Perf vs. SEC Relationship Comparisons 


In Figure 54 and Table 5, we see that 10 of the 13 relationships are stronger for non-defense pro¬ 
jects than they are for defense projects. The probability that 10 of the 13 would be stronger for the 
defense projects by chance alone is, again, unlikely (p <0.02 using a simple sign test). 
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Table 5: Stronger SE Process Areas 


Stronger Perfws. SEC Relationships 
for Defense Projects 

Stronger Perfvs. SEC Relationships 
for Non-Defense Projects 

• Project Monitoring and Controi 

• Trade Studies 

• integrated Product Teams 

• Total SE 

• Early SE 

• Verification 

• Project Planning 

• Requirements Development and Management 

• Product Integration 

• Validation 

• Configuration Management 

• Product Architecture 

• Risk Management 


3.2 Conclusions 

Our data shows that non-defense projects deploy less SE than defense projects, yet achieve better 
performance. We also find that the relationship between SE deployment and project performance 
for most SE process areas is stronger for non-defense projects than it is for defense projects. Thus, 
even though the non-defense projects deploy less SE, they gain more benefit from it, contributing 
to better project performance. 

Of particular significance are the SE process areas of Product Architecture, Requirements Devel¬ 
opment and Management, and Risk Management. These process areas show notably stronger rela¬ 
tionships for non-defense projects than for defense projects, as shown in Table 6. Understanding 
these areas in non-defense projects and adopting methodologies and activities for defense projects 
could produce significant improvements in the performance of defense-domain projects. 


Table 6: Notably Stronger Non-Defense Relationships 


Process Area 

Gamma 

(Defense) 

Gamma 

(Non-Defense) 

Product Architecture 

0.38 

0.54 

Requirements Development and Management 

0.46 

0.60 

Risk Management 

0.28 

0.40 

Prior Experience 

0.09 

0.54 


We also note several areas where activities in defense-domain projects show stronger relation¬ 
ships to project performance: 

1. Defense projects show a considerably stronger relationship between the performance of 
Trade Studies and project performance. 

2. The performance of defense projects appears to be less affected by project challenge for de¬ 
fense-domain projects than non-defense projects. 

Understanding the reasons behind these relationships in the defense domain and adopting them for 
non-defense projects could produce significant improvements in the performance of non-defense 
projects. 
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4 Next Steps 


Our data from the 2011 SE Effectiveness Survey identifies a number of SE process areas where 
SE activities in non-defense projects have impacts on project performance superior to those in 
defense-domain projects [Elm 2012b]. Likewise, it identifies several areas where SE activities in 
defense-domain projects have greater impacts on project performance than those in non-defense 
projects. Our data clearly shows these differences; unfortunately, it cannot explain why these dif¬ 
ferences exist. 

We plan to perform additional research to answer the following questions: 

1. What do non-defense projects do that makes their efforts addressing Product Architecture 
more effective than those of defense-domain projects? 

2. What do non-defense projects do that makes their efforts addressing Requirements Devel¬ 
opment and Management more effective than those of defense-domain projects? 

3. What do non-defense projects do that makes their efforts addressing Risk Management more 
effective than those of defense-domain projects? 

4. Why does prior experience in non-defense projects have more impact than for defense- 
domain projects? 

5. What do defense-domain projects do that makes their efforts addressing Trade Studies more 
effective than those of non-defense projects? 

6. What do defense-domain projects do that makes their performance less sensitive to project 
challenge than that of non-defense projects? 

The answers to these questions will enable the transplantation of best practices between the two 
domains to the benefit of both. 
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Appendix A Work Products Used to Assess SE Deployment 


• Alternative solutions 

• Baseline archives 

• Baseline audit records 

• Change control board 

• Commitment impacts 

• Concept of operations 

• Configuration baselines 

• Configuration item list 

• Cost and schedule baselines 

• Customer requirements list 

• Derived requirements list 

• Earned Value Management System (EVMS) data 

• EVMS updates 

• EVMS variance thresholds 

• Field problem assessments 

• Field problem reports 

• Integrated master plan 

• Integrated master schedule 

• Integrated product teams 

• Interface control documents 

• Interface descriptions 

• Peer review plan 

• Product architecture 

• Product integration process 

• Requirements acceptance criteria 

• Requirements allocations 


Requirements approval process 
Requirements configuration records 
Requirements impact assessments 
Requirements management system 
Requirements provider criteria 
Review of action items 
Review of issues 
Review of selection criteria 
Review process 
Risk list 

Risk mitigation plans 

Risk mitigation status 

SE master plan 

SE master schedule 

SE processes 

SE tracking records 

Technical approach 

Trade study records 

Use cases 

Validation criteria 

Validation procedures 

Verification criteria 

Verification entry and exit criteria 

Verification procedures 

Work breakdown structure 
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Appendix B Acronyms and Abbreviations 


EVMS 

EXP 

FFRDC 

IEEE 

INCOSE 

NDIA 

PC 

Perf 

PerfC 

PerfS 

PerfP 

SE 

SEC 

SEC-ARCH 

SEC-CM 

SEC-IPT 

SEC-PI 

SEC-PMC 

SEC-PP 

SEC-REQ 

SEC-RSKM 

SEC-Total 

SEC-TRD 

SEC-VAL 

SEC-VER 

SEI 

SPI 


Earned Value Management System 
prior experience 

federally funded research and development center 

Institute of Electrical and Electronic Engineers 

International Council on Systems Engineering 

National Defense Industrial Association 

project challenge 

project performance 

project cost performance 

project schedule performance 

project technical performance 

systems engineering 

systems engineering capability 

Product Architecture systems engineering capability 

Configuration Management systems engineering capability 

Integrated Product Teams systems engineering capability 

Product Integration systems engineering capability 

Project Monitoring and Control systems engineering capability 

Project Planning systems engineering capability 

Requirements Development and Management systems engineering capability 

Risk Management systems engineering capability 

total systems engineering capability applied to a project 

Trade Study systems engineering capability 

Validation systems engineering capability 

Verification systems engineering capability 

Software Engineering Insfitufe 

EVMS Schedule Performance Index 
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